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Foreword 


This report describes the work of the first winner of the IEEE Computer Society Software Pro- 
cess Achievement Award. This award was jointly established by the Software Engineering In- 
stitute (SEI) and the IEEE Computer Society to recognize outstanding achievements in 
software process improvement. It is given annually, if suitable nominations are received at the 
SEI on or before November 1 of any year. To obtain further information about the award, con- 
tact the award coordinator at the SEI. 

For the 1994 award, a total of 1 1 nominations were received and evaluated by a review com- 
mittee consisting of Vic Basili, Barry Boehm, Manny Lehman, Bill Riddle, and myself. Because 
of his intimate involvement with the winning organization, Vic Basili excused himself from the 
committee's decisions concerning this organization. 

As a result of reviewing the nominations and visiting several laboratories, the committee se- 
lected two finalists and the winner. The finalists are the process improvement staff of the Mo- 
torola Cellular Infrastructure Group and the software engineering process group of the 
Raytheon Equipment Division. The winner is the Software Engineering Laboratory (SEL) 
which is jointly operated by the NASA Goddard Space Flight Center, the Computer Sciences 
Corporation, and the University of Maryland. As a condition of the award, one or more repre- 
sentatives of the winning organization writes an SEI technical report on the achievement. This 
is that report. 

Many organizations have found that the lack of adequate data on the costs and benefits of 
software process improvement is a significant deterrent to their progress. This award thus em- 
phasizes both quantitative measures of process improvements as well as their significance 
and potential impact. While no single improvement approach will be appropriate for every or- 
ganization and while process improvement methods will evolve, the broad availability of such 
explicit improvement information should be of broad and general value. 

The granting of this award does not imply endorsement of any one improvement approach by 
the IEEE Computer Society or the SEI. The award committee does, however, endorse the ex- 
cellence of the work described in this technical report. We also feel it is particularly appropriate 
that the SEL win this first achievement award. This is both because of the technical signifi- 
cance of the SEL's work and because of its long history of leadership in software process re- 
search and improvement. The SEL was formed a full 10 years before the SEI was established 
and its distinguished record of contribution has had a profound impact on the work of software 
process professionals throughout the world. 


Watts S. Humphrey 
Chairman, Award Committee 
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Preface 


Since its inception, the SEL has conducted experiments on approximately 120 Flight Dynam- 
ics Division (FDD) production software projects, in which numerous software process changes 
have been applied, measured, and analyzed. As a result of these studies, appropriate pro- 
cesses have been adopted and tailored within the environment, which has guided the SEL to 
significantly improve the software generated. Through experimentation and sustained study 
of software process and its resultant product, the SEL has been able to identify refinements to 
its software process and to improve product characteristics based on FDD goals and experi- 
ence. 

The continual experimentation with software process has yielded an extensive set of empirical 
studies that has guided the evolution of standards, policies, management practices, technol- 
ogies, and training within the organization. The impacts of these process changes are evident 
in the resulting characteristics of FDD products. Over the period 1987 through 1993, the error 
rate of completed software has dropped by 75 percent; the cost of software has dropped by 
50 percent; and the cycle time to produce equivalent software products has decreased by 40 
percent. Additionally, the SEL has produced over 200 reports that describe the overall soft- 
ware process improvement approach and experiences from the experimentation process. 

This report describes the background and structure of the SEL organization, the SEL process 
improvement approach, and the experimentation and data collection process. Results of 
some sample SEL studies are included. It includes a discussion of the overall implication of 
trends observed over 17 years of process improvement efforts and looks at the return on in- 
vestment based on a comparison of the total investment in process improvement with the 
measurable improvements seen in the organization’s software product. 

As this report is a summary of many of the activities and experiences of the SEL which have 
been reported in more detail elsewhere, some of the material has been taken directly from oth- 
er SEL reports. These sources are listed in the Reference section of this document. Individual 
copies of SEL documents can be obtained by writing to 

Software Engineering Branch 

Code 552 

Goddard Space Flight Center 

Greenbelt, Maryland 20771 
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Software Process Improvement 
in the NASA Software Engineering Laboratory 


Abstract: The Software Engineering Laboratory (SEL) was established in 
1 976 for the purpose of studying and measuring software processes with the 
intent of identifying improvements that could be applied to the production of 
ground support software within the Flight Dynamics Division (FDD) at the 
National Aeronautics and Space Administration (NASA)/Goddard Space Flight 
Center (GSFC). The SEL has three member organizations: NASA/GSFC, the 
University of Maryland, and Computer Sciences Corporation (CSC). The 
concept of process improvement within the SEL focuses on the continual 
understanding of both process and product as well as goal-driven 
experimentation and analysis of process change within a production 
environment. 


1 Background 

1.1 SEL History 

The Software Engineering Laboratory (SEL) was created in 1 976 at NASA/Goddard Space 
Flight Center (GSFC) for the purpose of understanding and improving the overall software pro- 
cess and products that were being created within the Flight Dynamics Division (FDD). A part- 
nership was formed between NASA/GSFC, the University of Maryland, and Computer 
Sciences Corporation (CSC) with each of the organizations playing a key role: NASA/GSFC 
as the user and manager of all of the relevant software systems; the University of Maryland as 
the focus of advanced concepts in software process and experimentation; and CSC as the ma- 
jor contractor responsible for building and maintaining the software used to support the NASA 
missions. The original plan of the SEL was to apply evolving software technologies in the pro- 
duction environment during development and to measure the impact of these technologies on 
the products being created. In this way, the most beneficial approaches could be identified 
through empirical studies and then captured once improvements were identified. The plan was 
to measure in detail both the process as well as the end product. 

At the time the SEL was established, significant advances were being made in software de- 
velopment (e.g., structured analysis techniques, automated tools, disciplined management 
approaches, quality assurance approaches). However, very little empirical evidence or guid- 
ance existed for selecting and applying promising techniques and processes. In fact, little ev- 
idence was available regarding which approaches were of any value in software production. 
Additionally, there was very limited evidence available to qualify or quantify the existing soft- 
ware process and associated products, or to aid in understanding the impact of specific meth- 
ods. Thus, the SEL staff developed a means by which the software process could be 
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understood, measured, qualified, and measurably improved. Their efforts focused on the pri- 
mary goal of building a clear understanding of the local software business. This involved build- 
ing models, relations, and empirical evidence of all the characteristics of the ongoing software 
process and resultant product and continually expanding that understanding through experi- 
mentation and process refinement within a specific software production environment. 

1.2 SEL Process Improvement Strategy 

As originally conceived, the SEL planned to apply selected techniques and measure their im- 
pact on cost and reliability in order to produce empirical evidence that would provide rationale 
for the evolving standards and policies within the organization. As studies were performed, it 
became evident that the attributes of the development organization were an increasingly sig- 
nificant driver for the overall definition of process change. These attributes include the types 
of software being developed, goals of the organization, development constraints, environment 
characteristics, and organizational structure. This early and important finding provoked an in- 
tegral refinement of the SEL approach to process change. The most important step in the pro- 
cess improvement program is to develop a baseline understanding of the local software 
process, products, and goals. The concept of internally driven, experience-based process im- 
provement became the cornerstone of the SEL's “bottom-up” process improvement program. 

Incorporating the key concept of change guided by development project experiences, the SEL 
defined a standard paradigm to illustrate its concept of software process/product improve- 
ment. This paradigm is a three-phase model (Figure 1) which includes the following steps: 

1 . Understanding: Improve insight into the software process and its products by 
characterizing the production environment, including types of software devel- 
oped, problems defined, process characteristics, and product characteristics. 

2. Assessing: Measure the impact of available technologies and process 
change on the products generated. Determine which technologies are bene- 
ficial and appropriate to the particular environment and, more importantly, 
how the technologies (or processes) must be refined to best match the pro- 
cess with the environment. 

3. Packaging: After identifying process improvements, package the technology 
for application in the production organization. This includes the development 
and enhancement of standards, training, and development policies. 

In the SEL process improvement paradigm, these steps are addressed sequentially, and itera- 
tively, for as long as process and product improvement remains a goal within the organization. 

The SEL approach to continuous improvement is to apply potentially beneficial techniques to 
the development of production software and to measure the process and product in enough 
detail to determine the value of the applied technology within the specific domain of applica- 
tion. Measures of concern (such as cost, reliability, and cycle time) are identified as the orga- 
nization determines its major short- and long-term objectives for its software product. Once 
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these objectives are known, the SEL staff designs an experiment(s), defining the particular 
data to be captured and the questions to be addressed in each experimental project. This con- 
cept has been formalized in the goal-question-metric paradigm developed by Basili [Basili 84]. 


— 

PACKAGE 



Infuse improved (verified) process: 
• Standards and training 



r GOALS 

(e.g., reduce error rates) 

ASSESS 

(EXPERIMENT) 

Determine improvements to your business: 
• What impact does change have? 

UNDERSTAND 

Know your software business {process and product): 

• How do 1 do business today 

(e.g., standards and techniques used, % time in testing, module size) 

• What are my product characteristics? 

(e.g., error rates, productivity, complexity) 


Figure 1 : The SEL Process Improvement Paradigm 


All SEL experiments have been conducted in the production environment of the FDD at 
NASA/GSFC, which consists of projects that are classified as mid-sized software systems. 
The systems range in size from 10 thousand source lines of code (KSLOC) to over 1 .5 million 
SLOC. The original SEL production environment had approximately 75 developers generating 
software to support a single aspect of the flight dynamics problem. Over the years, the SEL 
operation has grown to include more extensive software responsibilities and, consequently, a 
larger production staff of approximately 300 developers and analysts. The SEL has been in 
continuous operation since 1976, and it will continue to operate as long as process and 
product improvement remain a priority within its software domain. 
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2 The SEL Organization 

The SEL comprises three partner organizations: the Software Engineering Branch at 
NASA/GSFC, the Institute for Advanced Computer Studies and Department of Comput- 
er Science at the University of Maryland, and the Software Engineering Operation at 
CSC. The total organization consists of approximately 300 persons. These personnel 
are divided into 3 functional components, not necessarily across organizational lines. 
The 3 functional areas are 

• Software development/maintenance 

• Process/product analysis 

• Database support 

The three components (developers, analysts, and database support) are separate, yet inti- 
mately related to each other. Each has its own goals, process models, and plans, but they 
share an overall mission of providing software that is continually improving in quality and cost 
effectiveness. The responsibilities, organizational makeup, and goals of the SEL components 
are discussed in the sections that follow. Figure 2 provides a graphic overview of their function 
and size, and Figure 3 depicts the difference in focus among the three groups. 


Developers 

(Source of Expertise) 


Process Analysts 

(Package of Experience for Reuse) 


Staff 

250-275 developers 

Typical Project Size 

100-300 KSLOC 

Active Projects 

6-10 

(at any given time) 

Project Staff Size 

5-25 people 

Total Projects 
(1976-1994) 

120 


Development 
measure for 

Staff 

10-15 analysts 

Function 

• Set goals/questions/metrics 

each project 


- Design studies/experiments 
• Analysis/research 



• Refine software process 

Refinements 


- Produce reports/findings 

to development 

Products 

300 reports/documents 

process 

m 

(1976-1994) 

NASA A CSC & University of Maryland 
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Developers 

Analysts 

Database Support Staff 

Focus and 
Scope 

• Specific software project 

• Domain (multiple projects) 

• Domain (multiple projects) 

Goals 

• Produce and maintain 
software 

• Satisfy user requirements 

• Analyze development and 
maintenance experience 
to define improvement 
process 

• Support developers 

• Archive, maintain, and 
distribute development 
and maintenance 
experience 

Approach 

• Use the most effective 
software engineering 
techniques, as provided 
by the analysts 

• Assess the impact of 
specific technologies 

• Produce models, 
standards, and training 
materials 

• Maintain a library of 
experiences, models, 
and standards 

Measure of 
Success 

• Validate and verify 
software products 

• Packaging and reuse of 
empirical software 
experience 

• Improved software 
products 

• Efficient processes for 
information retrieval 
(data models, reports) 


Figure 3: Focus of SEL Organizational Components 


2.1 Software Development/Maintenance 

The FDD development organization, comprising approximately 250-275 professional soft- 
ware developers, is responsible for development and maintenance of one segment of the 
ground support software used by GSFC. The majority of the software developers are CSC em- 
ployees under contract to NASA/GSFC; approximately 35 of the developers are employees of 
NASA/GSFC. SEL staff at the University of Maryland do not participate directly in the devel- 
opment or maintenance of flight dynamics software. 

For a typical project, FDD developers are provided a set of functional requirements for a mis- 
sion, from which they design, code, test, and document the software. The systems developed 
are primarily FORTRAN, non-real time, non-embedded, ground-based applications, and there 
are usually 4 or 5 projects in development at any one time. After the newly developed mission 
support software is tested and accepted, another team from this same organization takes over 
maintenance of the operational system. Approximately 50 percent of the development staff is 
allocated to software maintenance. 

The primary task of the development organization is to produce quality software on-time and 
within budget. They rely on another element of the SEL to carry out the analysis and packaging 
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of the process improvement studies. The development organization is not expected to pro- 
duce standards, policies, or training; nor are the developers expected to analyze data. The 
success of the development organization is measured by their ability to deliver a quality soft- 
ware product that meets the needs of the user. 

2.2 Process/Product Analysis 

The second major function within the SEL is analysis and process improvement. This effort is 
supported by personnel from all 3 member organizations: approximately 4 full-time people 
from NASA/GSFC; 5-10 individuals, each spending approximately 20 percent of their time, 
from the University of Maryland; and approximately 5-8 full-time people at CSC. This team de- 
fines studies to be conducted, analyzes process and products generated by the developers, 
and packages its findings in the form of updated standards, revised training programs, and 
new models specific to this development environment. All of the SEL analysts are experienced 
software engineers, many of whom have a number of years of experience in flight dynamics 
software development and/or maintenance. 

The analysts use information such as development environment profiles, process character- 
istics, resource usage, defect classes, and statistics to produce models of products and pro- 
cesses, evaluations, and refined development information. Their products include cost and 
reliability models, process models, domain-specific architectures and components, policies, 
and tools. 

The goal of the analysts is to synthesize and package experiences in a form useful to the de- 
velopment group. Their success is measured by their ability to provide in a timely way prod- 
ucts, processes, and information that can assist the developers in meeting their goals. 

2.3 Database Support 

The third function within the SEL is the data processing and archiving of the project’s experi- 
ences in the SEL’s measurement database. This is supported by approximately three full-time 
people at NASA/GSFC and approximately five full-time people at CSC. The database support 
staff collect the data that have been defined and requested by the analysts; assure the quality 
of those data; organize and maintain the SEL database; and archive the reports, papers, and 
documents that make up the SEL library (see Figure 2). 

The group includes both professional software engineers, who define and maintain the data- 
base, and data technicians, who enter the data, generate reports, and assure the quality of the 
information that is submitted to the SEL library. The goal of the database support organization 
is to manage the SEL measurement data and analysis products efficiently. Their success is 
measured by the efficient collection, storage, and retrieval of information, conducted in a way 
that doesn’t burden the overall organization with unnecessary activities and waiting periods. 
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3 The SEL Process Improvement Concept 


The SEL process improvement concept has matured over more than a decade, with the most 
significant changes to it being driven by experience at attempts to infuse process change and 
improvement within a production organization. The SEL improvement concept has occasion- 
ally been called an “experience factory” [Basili 92] and has also occasionally been termed a 
“bottom-up” software process improvement approach [McGarry 94]. The organization focuses 
on continually using experiences, lessons, and data from production software projects to en- 
sure that subsequent development efforts benefit, in terms of improved software products and 
processes, from the experience of earlier projects. The underlying principle of this concept is 
the reuse of software experiences to improve subsequent software tasks. This reuse of expe- 
rience is the driving element for change and improvement in the software process. 

3.1 Bottom-Up Improvement 

Although the term “ process improvement” is the term most commonly used to characterize the 
efforts of an organization to improve its software business, the SEL philosophy asserts that 
the actual goal of the organization is to improve the software product. The process improve- 
ment concept stems from an assumption that an improved process will result in an improved 
product. However, if a changed process has no positive impact on the product generated, then 
there is no justification for making change. A knowledge of the products, goals, characteristics, 
and local attributes of a software organization is needed to guide to the evolutionary change 
to process that focuses on the desired change to the product as defined by the goals of the 
organization. 

Two approaches to software process improvement have been developed and applied in the 
industry. The top-down approach (which is based on the assumption that improved process 
yields improved product) compares an organization’s existing process with a generally accept- 
ed high-quality standard process. Process improvement is then defined as the changes made 
to eliminate the differences between the existing process and the standard set of practices. 
This approach assumes that after change is made to the process, the generated products will 
be improved, or at least there will be less risk in the generation of new software. The most 
widely accepted and applied top-down model is the capability maturity model (CMM) [Paulk 
93], developed by the Software Engineering Institute (SEI). For the SEL, the CMM provides 
an excellent model for assessing process and for selecting potential process changes that 
mesh with local goals and needs. 

The SEL approach (sometimes called “bottom-up”) assumes that changes must be driven by 
local goals, characteristics, and product attributes. Changes are defined by a local domain in- 
stead of by a universal set of accepted practices. In this approach, software process change 
is driven by the goals of the particular development organization as well as by the experiences 
derived from that local organization. For example, an organization whose primary goal is to 
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shorten “time-to-ship” may take a significantly different approach to process change than 
would an organization whose primary goal is to produce defect-free software. 

The top-down approach is based on the assumption that there are generalized, universal prac- 
tices that are required and effective for all software development, and that without these prac- 
tices, an organization’s process is deficient. This paradigm has been accepted in many 
software organizations that have applied generalized standards, generalized training, and 
even generalized methods defined by an external organization (external to the developers) to 
all their software. This concept does not take into account the performance issues, problems, 
and unique software characteristics of the local organization. The implicit assumption is that 
even if an organization’s goals are being met and exceeded, if that organization does not use 
the commonly accepted practices, it has a higher risk of generating poor-quality products than 
an organization that adheres to the defined processes. The goals and characteristics of the 
local organization are not the driving elements of change. 

The underlying principle of the SEL approach is that “not all software is the same.” Its basic 
assumption is that each development organization is unique in some (or many) aspects. Be- 
cause of that, each organization must first completely understand its local software business 
and must identify its goals before selecting changes meant to improve its software process. If, 
based on that understanding, change seems called for, then each change introduced is guided 
by “experience” — not by a generalized set of practices. 

Neither the top-down approach nor the bottom-up approach can be effective if used in isola- 
tion. The top-down approach must take into consideration product changes, while the bottom- 
up approach must use some model for selecting process changes aimed at improving product 
characteristics. Each concept plays an important role in the goal of improving the software 
business. 

The CMM is designed as a framework that can help organizations better understand their soft- 
ware process and provide guidance toward reducing risk in software production. It provides an 
excellent procedure for identifying potentially beneficial additions to the organization’s software 
business practices. The SEL capitalizes on this paradigm to guide efforts at characterizing the 
software development process and to identify improvement areas. As a complement to the 
CMM (which provides specific approaches to assessing goals, products, and product at- 
tributes), the SEL model provides the tools for a complete and effective improvement program. 

Both approaches have similar difficulties capturing the image of the “local” software business, 
in defining exactly the scope or size of the local organization. Some judgment must be applied 
as to the components and boundaries of this single entity. SEL experience has shown that a 
smaller defined organization allows for a more detailed process definition and more focused 
refinements to the process. 
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3.2 Measurement 


The SEL approach uses a detailed understanding of local process, products, characteristics, 
and goals to develop insight. This insight forms the foundation of a measurable, effective 
change program driven by local needs. Because of this dependence on understanding the 
software within the subject environment, measurement is an inherent and vital component of 
the SEL approach: measurement of process and product from the start, measurement of the 
effect of process change on the product, and measurement of product improvement against 
the goals of the organization. The CMM provides guidance in building an understanding of 
software process within the development organization, but the SEL paradigm extends this 
concept to include product characteristics such as productivity, error rates, size attributes, and 
design characteristics. 

In the SEL approach, measurement is not viewed as a process element that is added as an 
organization matures, but rather as a vital element present from the start of any software im- 
provement program. An organization must use measurement to generate the baseline under- 
standing of process and product that will form the basis of the improvement program. 

The CMM includes the “software process assessment” tool, which is effective for generating 
baseline process attributes. The SEL’s bottom-up approach adds to those measures measure- 
ment of specific product characteristics, so that change can be effectively guided and observed. 

The SEL concept is driven by the principle that each domain or development organization 
must develop and tailor specific processes that are optimal for its own usage. Certainly, some 
processes and technologies are effective across a broad spectrum of domains (possibly even 
universal), but before a development organization settles on a particular process it must take 
the critical steps of understanding its software business and determining its goals. From there, 
change can be introduced in a structured fashion and its impact measured against the orga- 
nizational goals. 

3.3 Reuse of Experience 

Historically, a significant shortcoming in software development organizations has been their 
failure to capitalize on experience gained from similar completed projects. Most of the insight 
gained has been passively obtained instead of being aggressively pursued. Software devel- 
opers and managers generally do not have the time or resources to focus on building corpo- 
rate knowledge or planning organizational process improvements. They have projects to run 
and software to deliver. Thus, reuse of experience and collective learning must become a cor- 
porate concern like a business portfolio or company assets. Reuse of experience and collec- 
tive learning must be supported by an organizational infrastructure dedicated to developing, 
updating, and supplying upon request synthesized experiences and competencies. This orga- 
nizational infrastructure emphasizes achieving continuous sustained improvement over iden- 
tifying possible technology breakthroughs. 


CMU/SEI-94-TR-22 


11 


The SEL represents this type of organizational element. It is focused solely on reuse of expe- 
rience and software process improvement with the goal of improving the end product. Because 
these activities rely so significantly on actual software development experiences, the develop- 
ers, analysts, and database support staff organizations, while separate, are intimately related 
to each other. Developers are involved in process improvement activities only to the extent 
that they provide the data on which all process change is based. Process/product analysts and 
database support personnel are dedicated to their process improvement responsibilities and 
are in no way involved in the production of software product. Additionally, the SEL research/- 
database support teams have management and technical directors separate from the devel- 
opment projects. This ensures continuity and objectivity in process improvement activities and 
the availability of resources for building, maintaining, and sustaining the process improvement 
program. 
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4 SEL Experimentation and Analysis 


Each production project in the FDD is considered an opportunity for the SEL to expand its 
knowledge base of process understanding and improvement. There are typically 4 or 5 
projects under development at any one time, and an additional 1 5 to 20 projects in the main- 
tenance phase. All of the projects in the FDD environment are considered experiments, and 
the SEL has completed over 120 project studies over the years. For each of these projects, 
detailed measurements were provided toward the end goal of analyzing the impact that any 
change to software process had on the resultant software product. 

When research in the production environment is being planned, the following activities occur: 
the SEL analysis team defines a set of goals that reflects current goals in process/product im- 
provement and writes an experiment plan in which required data are identified and experimen- 
tal processes are outlined; a SEL representative is assigned to the project/experiment; and 
technology/process training needs are assessed. SEL software development/maintenance 
project personnel then provide the requested information (defined in the experiment plan) to 
the SEL database support staff who add it to the database for access by the analysts conduct- 
ing the experiment. These SEL activities are described in the sections that follow. 


4.1 Defining Experiments 

SEL analysts identify software process modifications that they hypothesize are likely to im- 
prove the resultant product, and then design an experiment to test the hypothesis. As experi- 
ments are being defined, the analysts consult the development team to determine if proposed 
changes (such as applying a particular technique) could be studied on a project without undue 
risk. Even if risk is significant, a team may be willing to try the new process provided a contin- 
gency plan is developed to assure that a disaster can be avoided. It is important that the de- 
velopment team be factored into decisions on the proposed changes and that their full support 
is obtained. 

Once a project is identified and a modified process is selected, an experiment plan is written 
describing the goals, measures, team structure, and experimental approach. A sample SEL ex- 
periment plan is included in Appendix A. If the study is very small (e.g., collect inspection data 
to measure the cost of software inspections), a formal experiment plan may not be written. 

The basic project/experiment information is then provided to the SEL database support group 
so that project names, subsystem names, personnel participating, and forms expected can be 
logged, and the database can be readied for data entry. 

Once an experiment is defined and the study objectives have been agreed upon with the 
developers, a representative from the analysts is assigned to work directly with the 
development team for the duration of the project. This representative keeps the development 
team informed of experimental progress, provides information on the particular process 
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changes being applied, and answers any questions the development team may have with 
regard to SEL activities. The SEL representative does not manage or direct the development 
project in any way. The SEL representative attends reviews and development status meetings 
and looks at measurement data collected. At the conclusion of the project, the SEL 
representative also writes a section for inclusion in the project’s development history report 
which discusses the experimental goals and results. 

For most projects, the experiment being conducted does not have a significant impact on the 
development procedures and typically does not involve major changes to the technologies be- 
ing applied. If there is a more significant change (e.g., using Ada, applying Cleanroom tech- 
nique, or using inspections with a team unfamiliar with the technology), the analysts arrange 
for training for the development team. For example, when the SEL studied Cleanroom tech- 
nique on one project, approximately 40 hours of training in the technique were provided to the 
first development team using it in this environment [Green 90]. 

4.2 Collecting Measures 

In support of the SEL experiments, technical and management staff responsible for software 
development and maintenance provide the requested measurement data. Although the types 
of data requested may vary from project to project to satisfy the requirements of particular ex- 
periments, the core set of information is invariant. Basic data are collected from every project, 
including effort, defects, changes, project estimates, project dynamics (e.g., staffing levels), 
and product characteristics. These data are provided on data collection forms. Figures 4 and 
5 are samples of the forms used to report effort data and defect/change data. Details of the 
core measures used, as well as the measurement program in general, can be found in the 
Software Measurement Guidebook [Bassman 94]. The full set of data collection forms and pro- 
cedures can be found in the Data Collection Procedures for the Software Engineering Labo- 
ratory Database [Heller 92]. 
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SECTION A: Total Hours Spent on Project for the Week: 


SECTION B: Hours by Activity 

(Total of hours in Section B should equal total hours in Section A.) 


Activity Activity Definitions 

Predesign Understanding the concepts of the system. Any work prior to the actual design (such as requirements analysis). 

Create Design Development of the system, subsystem, or components design. Includes development of PDL, design diagrams, etc. 

Read/Review Hours spent reading or reviewing design. Includes design meetings, formal and informal reviews, or walkthroughs. 
Design 

Write Code Actually coding system components. Includes both desk and terminal code development. 

Read/Review Code reading for any purpose other than isolation of errors. 

Code 

Test Code Testing individual components of the system. Includes writing test drivers. 

Units 


Debugging 


Integration 

Test 


Hours spent finding a known error in the system and developing a solution. Includes generation and execution oi tests associated with 
finding the error. 

Writing and executing tests that integrate system components, including system tests. 


Acceptance Running/supporting acceptance testing. 

Test 

Other Other hours spent on the project not covered above, includes management, meetings, training hours, notebook, system description, user’s 

guides, etc. 


SECTION C: Effort on Specific Activities (need not add to Section A) 

(Some hours may be counted in more than one area; view each activity separately.) 


Rework: Estimate of total hours spent that were caused by unplanned changes or errors. Includes effort 

caused by unplanned changes to specifications, erroneous or changed design, errors or 
unplanned changes to code, changes to documents. (This includes all hours spent debugging.) 

Enhancing/Refining/Optimizing : Estimate of total hours spent improving the efficiency or clarity of design, code, or documentation. 

(These are not caused by required changes or errors in the system.) 




Documenting. 


Hours spent on any documentation on the system. 

(This includes development of design documents, Prologs, in-line commentary, 
test plans, system descriptions, user's guides, or any other system documentation.) 

Hours spent in an effort to reuse components of the system. 

(This includes effort in looking at other system(s) design, code, or documentation. 
Count total hours in searching, applying, and testing.) 




| For Librarian s Use Only | 

Number 


Date: 


Entered by: 


Checked by: 



Figure 4: Effort Data Collection Form 
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CHANGE REPORT FORM 


Approved by: 
Date: 

Section A — Identification 

Describe the change (what, why, how): 


Name: 

Project 


Effect: What components are changed? 


Pteli* 

1 Name 

1 Version j 











(Attach list if more space is needed ) 


Location of developer's source files 


Need for change determined on: 

Change completed (incorporated into system): 

Effort in person time to isolate the change (or error): 







rvontn j 

1 

year 




Effort (What additional components were examined in 
determining what change was needed?): 


Check here if change involves Ada components. 
(If so, complete questions on reverse side.) 



Effort in person time to implement the change (or correction): 




Section B — All Changes 

Type of change (check one): 


Q Error correction 

Q Planned enhancement 

0 Implementation of requirements change 

Q Improvement of clarity, maintainability, 
or documentation 
Q Other 


Q Improvement of user services 
[J Inserforvkletetion of debug code 
Qj Optimization of tim^space'accuracy 
Q Adaptation to environment change 


Section C — For Error Corrections On 


| Source of Error 

[ (check one) 

□ 

Requirements 

□ 

Functional specifications 

□ 

Design 

□ 

Code 

□ 

Previous change 


Class of Error 
(cheek most applicable’) 


j | initialization 

'J Logic/control structure (eg., flow of control hcorrect) 
n ,n1ef,ac e (internal) (module-to-module conYixjn baton) 
Q Interlace (external) (module-to-extemal communication) 
| j Data value or structure (e g , wrong variable used) 

| | Computational (e g., error in math expression) 

•if two are equally appficable. check the one hither on the list 


m 

m 

ra 


Effects of Change 

Was the change or correction to one and only one component? 
(Must match Effect in Section A.) 


Commission error (i.e., something incorrect was included) 


Transcripton (clerical) error 


Characteristics 


For Librarian s Use Only 

Number 


Date 


Entered by 


Checked by 



^1*1 Omission error (i.e., something was left out) 
Commission error 

(i.e., something incorrect was included) 

■D 

1 | j Transcription (clerical) error 


Figure 5: Defect/Change Data Collection Form 

. K, IO«)l 
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As the developers/maintainers complete the forms, they submit them to the database support 
personnel who assure the quality of the information by checking the forms and data for con- 
sistency and completeness. When data are missing (e.g., if an expected form is not submit- 
ted), the developer is informed of the discrepancy and is expected to provide or correct the 
data. Database support staff then enter the data in a central database and perform a second 
quality assurance step by checking for data entry errors by comparing the database informa- 
tion against the original paper forms. 

In addition to the forms that are completed by the developers and managers, several tools are 
used to gather automatically information such as source code characteristics (e.g., size, 
amount of reuse, complexity, module characteristics) or changes and growth of source code 
during development. Database support personnel execute the tools to gather these additional 
measures, which are then entered in the SEL database. 

Additionally, subjective measures are recorded on the development process. These data are 
obtained by talking with project managers and by observing development activities. Data such 
as problem complexity, adherence to standards, team experience, stability, and maturity of 
support environment are captured at the termination of each project. (See [Bassman 94] for 
details on these measures.) 

Figure 6 depicts the life-cycle phases during which the core SEL measures are collected. Each 
project provides these data and may provide additional measures required for the specific ex- 
periment in which it is participating. 

4.3 Analyzing Data 

The analysts use these data together with information such as trend data, previous lessons 
learned, and subjective input from developers and managers, to analyze the impact of a spe- 
cific software process and to build models, relations, and rules for the corporate memory. As 
specific processes are studied (such as inspections, Cleanroom), the analysts, joined by will- 
ing participants from the development organization, complete analysis reports on the study 
and may even prepare a paper or report for publication in the open literature. Development 
team participation is strictly voluntary in this step, as the analysts are ultimately responsible for 
producing the report. 

As the project information becomes available, the analysts use it not only to assess particular 
processes, but also to build models of the process and product so that the experiences of each 
development effort can be captured and applied to other projects where appropriate. Data are 
used to build predictive models representing cost, reliability, code growth, test characteristics, 
changes, and other characteristics. The analysts also look at trends and processes applied to 
determine whether or not any insight can be gained from data describing particular methodol- 
ogies used during development or maintenance. 
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Process 

• methods 

• tools 
•(etc.) 


Product 

• size 
•cost 

• (etc.) 


Dynamics 

(growth, changes, ...) 


Errors/changes 
(unit test to delivery) 


Maintenance errors 


Project estimates 
(size, cost, dates, reuse) 




Development effort 
(tracked by time and by activity) 



Maintenance effort 

A 

Functional 

requirements 

received 

Requirements 

Analysis 

Design 

Code 

Test 

1 

Acceptance 

A 

Begin 

maintenance 
and operation' 

Maintenance 


Figure 6: SEL Core Measures 


One of the most important facts that the SEL has learned from its experience with analysis of 
software data is that the actual measurement data represent only one small element of exper- 
imental software engineering. Too often, data can be misinterpreted, used out of context, or 
weighted too heavily even when the quality of the information may be suspect. Having learned 
from its extensive data analysis experience over the years, the SEL now follows these key 
rules: 


• Software measures will be flawed, inconsistent, and incomplete; the analysis must 
take this into account. Do not place unfounded confidence in raw measurement data. 

Even with the extensive quality-assurance process and the rigor of the software 
measurement collection process in the SEL, the uncertainty of the data is still quite 
high. An analyst must consider subjective measures, qualitative analysis, definition 
of the context, and an explanation of the goals. If one merely executes a high 
number of correlation analysis studies on a high number of parameters, chances are 
that some (possibly very questionable) statistic will appear. Extreme caution must be 
applied when using software measurement data, especially when the analyst is not 
intimately familiar with the environment, context, and goals of the studies. 
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• Measurement activity mst not be the dominant element of software process 
improvement; analysis is the goal. 

When the SEL began to study software process, the overhead of the data collection 
process dominated the total expenditures for experimental activities. As the SEL 
matured, it found that the successful analysis of experiments should consume 
approximately three times the amount of effort that data collection activities require. 
This ratio was attained through a gradual cutback in data collection to where the only 
information requested (beyond the core measures) was that which could be clearly 
defined as relevant to the goals of a particular experiment. 

• Measurement information must be treated within a particular context; an analyst 
cannot compare data where the context is inconsistent or unknown. 

Each set of measurement data that is archived in the SEL database represents a 
specific project, with unique characteristics and unique experimental goals. These 
goals may have significantly influenced the process used, the management 
approach, and even the general characteristics of the project itself. Without 
knowledge of the context in which the data were generated and the overall project 
goals as well as process goals, significant misinterpretations of the data can result. 


4.4 Improving Process 

Measurement activities represent a relatively small element of the overall process improve- 
ment task. Results of analysis of experimental data must be judiciously applied toward opti- 
mizing the software development and maintenance process. The experimental software 
engineering results are captured both in studies as well as in refined processes available to 
the production personnel. The SEL packages its analysis results in the form of updated stan- 
dards, policies, training, and tools. This packaging facilitates the adoption of revisions to the 
standard processes on ongoing and future software projects. 

The SEL conducts three general types of analysis, all of which are active continually in the en- 
vironment. They include 

• Pilot studies of specific techniques and technologies on a project or set of 
projects [e.g., Cleanroom impact on design, impact of object-oriented design 
(OOD) on code reuse, impact of inspection on coding errors]. 

• Studies of completed projects for development and refinement of local 
process and product models (e.g., cost models, error characteristics, reuse 
models). 

• Trend analysis of completed projects to track the impact of specific process 
changes on the environment as a whole (e.g., tailored Cleanroom, OOD, 
software standards). 
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All of the analyses are dependent on the project measures and all require a thorough under- 
standing of context, environment, goals, problem complexity, and project characteristics to be 
able to derive results that can be fed into the overall process improvement program. 

A study of a specific process or technique is usually termed a “pilot study.” Although these 
studies often occur in the university environment, they are also conducted on production 
projects where some risk can be tolerated. These projects are testing new and unfamiliar tech- 
niques to determine their value in the production environment and to determine whether more 
extensive studies would be beneficial. On pilot projects, the analyst typically analyzes each 
phase of the project in detail and reports back to the development team the intermediate re- 
sults as the project progresses toward completion. In general, the SEL conducts no more than 
two pilot studies at any one time because the amount of analysis and reporting is so extensive. 
These studies normally yield multiple reports and papers that look at every aspect of the im- 
pact of the new technology, make recommendations for tailoring, and project the value of the 
enhanced process in an expanded application. 

The second class of study involves multiple projects, where the goal is to expand and update 
the understanding of process and product attributes. Cost models are enhanced, error at- 
tributes are studied, and relations between process and product characteristics are analyzed 
for classes of projects. These studies normally do not use data from projects under develop- 
ment, but focus on completed projects. This type of analysis requires not only the archived 
measurement data, but also a detailed knowledge of each project’s context (including goals, 
processes used, problem complexity, size, and other product attributes). Trends in software 
quality, productivity, as well as profiles of the software product are produced so that specific 
needs and potential process enhancements can be identified. 

Trend analysis also looks at multiple completed projects. The goal of these studies is to deter- 
mine the appropriate application of evolving technology and methods within the environment 
as a whole, or at least for a specific class of projects. After pilot projects have been completed 
and appropriate tailoring or enhancement of process changes have been made, additional 
projects apply the tailored process. The additional application of the methods may involve only 
a single element of the originally defined process. For instance, although the Cleanroom meth- 
odology includes specific techniques for design, testing, management, implementation, and 
the inspection process, it may turn out that only the testing and implementation techniques are 
appropriate for further application. Once it is determined which process changes are appropri- 
ate for a broader class of projects (or possibly the entire development environment), these el- 
ements of the process are incorporated into the software standards and policies. Additionally, 
the training program may be updated to reflect the refined process. (See the discussion of 
packaging in Chapter 5 for a detailed description of the SEL training program.) 
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5 SEL Experiences: Understanding, Assessing, and 
Packaging 

The SEL paradigm has been applied on approximately 120 production projects in the FDD. 
Each project has provided detailed measurement data for the purpose of providing more in- 
sight into the software process, so that the impact of various software technologies could be 
empirically assessed. Projects have ranged in size from 10 KSLOC to 1.5 million SLOC, with 
the majority falling in the 100-250 KSLOC range. Appendix B lists the general characteristics 
of the projects that have been studied in the SEL to date. All of the information extracted from 
these development and maintenance projects is stored in the SEL database and used by the 
analysts who study the projects and produce reports, updated standards, policies, and training 
materials. 

During the understanding phase of the SEL paradigm, the goal is to produce a baseline of de- 
velopment practices and product attributes against which change can be measured as pro- 
cess modifications are applied. Additionally, the understanding process generates the models 
and relations used to plan and manage the development and maintenance tasks. The goal of 
the assessing or experimental phase is to determine the impact of specific process changes 
on the overall goals of the organization. In the packaging phase of the paradigm, those prac- 
tices that have proven measurably beneficial are incorporated into the organization’s stan- 
dards, policies, and training programs. 


5.1 Understanding 

Probably the most critical element of the SEL’s process improvement program is the under- 
standing step — where the only goal is to gain insight into the local software business. This first 
step cannot provide the justification for claiming that one process is better than another, but 
instead yields a baseline of the characteristics of the software, including both process and 
products, upon which change and meaningful comparison can be based. 

Although the initial plan was to begin experimenting with various techniques, the SEL soon 
learned that without a firm, well-understood baseline of both process and product characteris- 
tics, valid experimentation was impossible. In order to build this understanding, information 
gathered from the first 5-10 projects was primarily used to generate models, relations, and 
characteristics of the environment. These models and their understanding proved to be a sig- 
nificant asset to the management, planning, and decision making needed for effective soft- 
ware development. 

The understanding process, begun with those first 5-10 projects, continues today on all 
projects. The various models are continually updated as the process is better understood, and 
as new technologies and methods change the way the SEL views software development. Figure 
7 lists 1 1 projects active between 1985 and 1990 that were included in the early SEL baseline. 
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Project 

Start Date 

End Date 

GROSIM 

8/85 

8/87 

COBSIM 

1/86 

5/87 

GRODY 

9/85 

7/88 

COBEAGSS 

6/86 

7/88 

GROAGSS 

8/85 

4/89 

GOESIM 

9/87 

7/89 

GOFOR 

6/87 

9/89 

GOESAGSS 

8/87 

11/89 

UARSTELS 

2/88 

12/89 

GOADA 

6/87 

4/90 

UARSAGSS 

11/87 

9/90 


Figure 7: SEL Baseline (1985-1990) 


By examining the effort data of these projects, the SEL built its baseline of software cost ex- 
penditures by phase and by activity. This is some of the most basic, yet often overlooked, in- 
formation for software environments. By looking at a series of projects, a simple model of effort 
distribution can be built to depict the cost of design, code, test, and other activities. Such data 
are accumulated weekly from all developers, managers, and technical support using a data 
collection form. The form captures effort expended on software design, testing, coding, and 
the amount of time spent on code reading vs. code writing. (See Figure 4 for a sample effort 
data collection form.) 

Figure 8 illustrates distribution for the effort data based on the projects in this baseline. These 
data represent 1 1 projects over 5 years, consuming a total of approximately 65 staff-years of 
effort. The data show that approximately 1/4 of the cost of producing the software is spent on 
activities other than designing, coding, or testing. This “other” activity includes meetings, 
travel, reviews, training, etc. The SEL has found that the value for “other activities has 
remained almost constant for the entire time the SEL has been closely monitoring projects; in 
fact, it has increased slightly over time instead of decreasing as SEL staff first expected that it 
would. This time represents an important component for project budgets, one that is often 
overlooked by managers who lack a thorough understanding of their baseline process. One 
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surprising observation has been that the basic characteristics of this environment do not 
radically change from year to year even with continuous modifications being made to the 
underlying processes. The profile of the software environment changes very slowly. In Figure 
8, the data are represented in two ways: 

• One representation is effort by phase, where the total hours reported each 
week are attributed to the phase that the project is currently executing: i.e., 
designing from start through review and acceptance of design, coding from 
start through beginning of system testing, and testing from the start through 
system delivery. These data require only that the phase dates be known and 
that the total hours worked each week be reported by the development 
staff. 

• The second representation is effort by activity, where weekly information is 
broken down to the particular activity that the programmers were performing 
during that week. For example, they may report design hours even though 
the project was well into the coding phase. This modeling of the data provides 
a more accurate view of project interactions, as compared to the model that 
relies on (somewhat arbitrary) phase dates often set before project initiation. 


Acc Test 15% 



Effort Distribution by Life-Cycle Phase Effort Distribution by Activity 

* code writing: 85% 
code reading:15% 

Figure 8: Effort Distribution by Phase and Activity 
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Along with cost and schedule, reliability and correctness of the resulting code are considered 
attributes of interest to management. These attributes also contribute to the expanding 
understanding of the software process and product in the environment. The SEL captures 
these attributes by collecting defect data. The SEL defined its own classes of errors to ensure 
internal consistency in the data. Types of errors include 

• Computational errors — improper calculations within the source program, 
such as writing the wrong form of an expression. 

• Interface errors — include both internal and external errors and represent 
invalid information (e.g., wrong data) being passed between modules, such 
as in a subroutine call. 

• Logic/control errors — errors in flow control in a program, such as incorrect 
branches as the result of evaluating an if-statement expression. 

• Initialization errors — improper settings of the initial value of variables. 

• Data errors— wrong variable used in a calculation. 

The SEL continually collects error data (starting when unit test is completed and continuing 
through delivery of the software and during maintenance) so that it is possible to continually 
understand the numbers and types of errors occurring in the software. This information is as 
important as the effort data. Together, they constitute two of the most critical core measures 
that the SEL has found. On maintenance projects, defect data are collected on a modified form 
which the SEL developed in 1 990 when the organization became responsible for software 
maintenance as well as development. 

Over 2000 errors were classified and studied from the projects in the 1 985-1 990 baseline. The 
error class distribution as well as the origin of errors (i.e., during what phase/activity the defect 
entered the software) are shown in Figure 9. 

An earlier SEL study of errors provides an example of how models of software characteristics 
can be developed. By tracking five projects of similar complexity and size, the uncovered er- 
rors showed a decreasing step function for their rate of detection during sequential phases of 
the projects. From these data and trends, the SEL developed an internal model of expected 
error occurrence and detection rates for its class of software (see Figure 10). More recent 
studies show that the step function is still present, although the error rates have decreased 
significantly. 
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Initialization 16% 


Logic/Control 20% 



Requirements 20% 


Design 30%. 



Previous change 1 0% 


Classes of Errors* 


Origin of Errors* 


• Data from approximately 1 1 projects over 5 years (over 4000 errors sampled) 


Figure 9: SEL Error Characteristics 


▲ ▲▲ 


Code/Test System Test 


Acceptance _ 

Test Operations 


Based on five similar projects in SEL 


Figure 10: Error Detection Rate Model 
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In addition to effort and defect data, other parameters are useful for developing a total under- 
standing of the local environment. By counting defects found during the development of the 
software, then counting defects found during the operation and maintenance phases, the SEL 
developed a general understanding of the overall reliability of the software. Models of charac- 
teristics such as defects, change rate, effort distribution, and documentation size all provide 
useful information toward the development of improved models of software. This leads toward 
the capability of engineering the software process with well-understood relations, models, and 
rules. Using a sampling of projects developed during the early years of the SEL (late 1970s to 
mid-1980s), a set of models and relations was produced which was used as the baseline for 
planning, managing, and observing change over time (see Figure 11). One of the more sur- 
prising observations was that, after years of operation, the models changed very slowly — even 
with the significant technology and process changes introduced over time. Figure 1 2 describes 
the characteristics of another set of software projects active during the latter part of the 1 980s. 
The differences between this and the earlier models/relations are surprisingly small, but there 
is change. 



Figure 11: Initial SEL Models/Relations 


Of all the models and relations that the SEL has developed during the understanding phase, 
the most useful for project planning and management and for observing change have been 

• Effort distribution (cost characteristics). 

• Error characteristics (numbers, types, origins). 

• Change and growth rates (of the source code during development). 
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The first two of these have been described in some detail in this section. These very basic 
pieces of information are being collected continually; they are used to observe change and im- 
provement and to assess process impact. 



Figure 12: More Recent SEL Software Characteristics (late 1980s) 


5.2 Assessing 

After establishing a baseline of process, product, and environment characteristics and deter- 
mining organizational goals, the next step in applying the SEL paradigm is to assess the value 
of any process change. In the SEL, these assessments are called “experiments,” and each 
project that is developed in the production environment is viewed as an experiment. Some of 
the studies are meant only to establish models of process or product, while other experiments 
are designed to evaluate the impact that a significant process change may have on the local 
software business — both process and product. Some of the experiments do not make overt 
changes to the established development process in the SEL, but are monitored mainly to es- 
tablish the baseline understanding of the process. Additionally, some technologies require 
multiple projects to be completed before the impact of the change can be fully understood and 
before recommendations can be made for tailoring the process for local use. 

It is perhaps useful to comment here on the relationship between the SEL paradigm and the 
CMM. Others have contrasted the two as alternative approaches to solving the same prob- 
lem — process improvement. However, the SEL’s understanding/assessing/packaging model 
and the CMM are quite distinct. The SEL uses a specific process model to drive the organiza- 
tion. Via this model, decisions are made as to how to manage a software development project. 
Other development models certainly exist, such as a straightforward “waterfall” or MIL-STD- 
2167A. On the other hand, the CMM is an assessment model, and can be used to assess the 
process improvement of the SEL as well as other organizational structures. Thus the two con- 
cepts are actually complementary and can be used separately or jointly in any organization. 
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The structure of the SEL, as a partnership of GSFC, CSC, and the University of Maryland, has 
permitted a wide variety of experiments to be conducted, maximizing the skills and resources 
of each of the contributing organizations. Experiments have ranged across numerous technol- 
ogies, from minor process change (e.g., adding code-reading techniques to measure resulting 
error rates) to major process change (e.g., object-oriented design, Cleanroom, Ada). Through 
the experimentation process, the SEL has gained broad insight into the impacts of these tech- 
nologies and processes and has reported extensively on its findings. Some representative 
studies are discussed in the paragraphs to follow. They include assessments of 

• Design approaches 

• Testing techniques 

• Cleanroom methodology 

• Ada/OOD 

• Independent verification and validation (IV&V) 

5.2.1 Studies of Design Approaches 

Some studies require only an understanding of the current development environment. These 
are low-impact studies that can be undertaken with little risk to projects under development. 
The following design study is one such experiment. 

In 1985, several experiments were conducted to determine the value of various design char- 
acteristics on the quality of the end product. This particular study used available information 
already being captured from development projects; there was no need to retrain the develop- 
ment personnel in particular design techniques. The goal was to determine if the “strength and 
coupling” criteria described by Constantine and Meyers [Stephens 74] could be used as a pre- 
dictive metric to determine the reliability of software. 

A set of 453 software modules was selected from 9 completed projects for which detailed mea- 
surement information existed. The measures included design characteristics, number of de- 
fects found in the modules, and module size. This study was described in detail in a paper 
presented at the 1985 International Conference on Software Engineering [Card 85]. 

Strength was measured by the number of functions performed by an individual module, as de- 
termined by the authoring programmer. The 453 modules were classified in the following way: 

• 90 modules were of low strength and averaged 77 executable statements. 

• 176 modules were of medium strength and averaged 60 executable statements. 

• 187 modules were of high strength and averaged 48 executable statements. 

As a control, module size was also used. Small modules had up to 31 executable statements; 
medium-sized modules had up to 64 executable statements; and large modules had more 
than 64 executable statements. Error rates were classified as low (0 errors/KLOC), medium 
(<3 errors/KLOC), and high (> 3 errors/KLOC). 
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In analyzing error rates for these modules, strength proved an important criterion for determin- 
ing error rates (see Figure 13) and proved more effective than simply using size as a predictor 
for defects. For example, 44 percent of the low-strength modules had high error rates; for high- 
strength modules, error rates ranged from 44 percent to only 20 percent. On the other hand, 
using size as a predictor of error, 27 percent of large modules were error prone while 36 per- 
cent of small modules were error prone, indicating that high strength was a good indicator of 
fewer defects. 

Using all of the data available for the study, the SEL’s baseline understanding for strength be- 
came: 

• Good programmers tend to write high-strength modules. 

• Good programmers tend not to show any preference for particular module size. 

• Overall, high-strength modules have a lower fault rate and cost less than low- 
strength modules. 

• Fault rate is not directly related to module size. 



High-Strength Modules Medium-Strength Modules Low-Strength Modules 


1 Error rates I 

medium 

£ 3 errors/KLOC 

high 

> 3 errors/KLOC 


Figure 13: Fault Rate for Classes of Module Strength 
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5.2.2 Studies of Testing 

Some studies are best carried out in small controlled environments. Using the university 
environment as an initial testing laboratory is useful for these studies. After validating the 
results in the university environment, the concept can be applied in an operational setting. The 
following testing experiment is an example of that approach. 

Reliability of the software produced is of continuing concern to the SEL. The goal of one study 
was to evaluate several testing techniques in order to determine their effectiveness in discov- 
ering errors. The techniques evaluated in this experiment were 

• Code-reading of the source program by programmers other than the authors. 

• Functional (i.e., black box) testing of the source program to the specifications 
(i.e., in-out behavior) of the program. 

• Structural (i.e., white box) testing by developing test cases that execute 
specific statement sequences in the program. 

Initially, a study was performed at the University of Maryland using 42 advanced software en- 
gineering students. Based upon positive results of this initial study, 32 programmers from 
NASA and CSC were recruited. All knew all 3 techniques, but were most familiar with the func- 
tional testing approach generally used at NASA. Three FORTRAN programs were chosen 
(ranging from 48 to 144 executable statements containing a total of 28 faults). All 32 program- 
mers evaluated the 3 programs using a different testing technique on each program. 

The main results of this study can be summarized as follows: 

• Code reading was more effective at discovering errors than was functional 
testing, and functional testing was more effective than structural testing (see 
Figure 14). 

• Code reading was more cost effective than either functional testing or 
structural testing in number of errors found per unit of time (see Figure 15). 
Structural testing and functional testing had about the same costs. 


Number of Faults Detected 


3.3 


Structural 

Code reading uncovered more errors than other methods; functional testing uncovered more errors than structural testing (a < .005). 

While different quantities of faults were detected in each program, the percentage of faults detected/program was the same. 

Advanced students uncovered more faults than other students (a < .005); intermediate students uncovered the same amount of faults as junior students. 
Percent faults detected correlates with percent felt by tester to have been uncovered R = .57 (a < .001). 

Figure 14: Fault Detection Rate by Testing Method 
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Cost-Effectiveness {number of faults detected/efforl) 



Reading Functional Structural 


Code reading was more cost-effective than other methods [(a < .005), est +1.5(.4)]. 

There was a different overall detection rate for one program. 

Techniques did not vary in total detection time. 

Figure 15: Cost of Fault Detection by Testing Method 


The study also produced some interesting results concerning programmer expertise and the 
discovery of faults. Space does not permit a full explanation here (see [Basili 87] for further 
details), but the results can be summarized as follows: 

• The FORTRAN program built around abstract data types had the highest 
error discovery rate. This was an early indicator of the value of OOD. 

• More experienced programmers found a greater percentage of the faults 
than less experienced programmers. 

• Code reading and functional testing found more omission and control faults 
than structural testing. Code reading found more interface faults than the 
other two techniques. 

This study, besides providing an assessment of the value of each of the testing techniques, 
adds to our understanding of the underlying baseline technology for later experiments. 

5.2.3 Studies with Cleanroom 

The following study of Cleanroom software development is an example of the use of pilot 
studies of new processes that pose great risks to the development organization. In this case, 
the method was studied for several years at the University of Maryland before being testing in 
the SEL operational environment. 

Reliability and defect rates have always been important components of understanding the en- 
vironment. The Cleanroom technique, developed by Harlan Mills of IBM, proposed to radically 
alter how programs are developed in order to affect these rates. The SEL looked at Cleanroom 
as another process that might significantly improve their development process. 
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The idea behind Cleanroom is relatively simple. After a programmer implements a function, 
the programmer must verify that the function meets its specification, rather than relying on unit 
testing to show that it apparently works. Cleanroom, then, has the following attributes: 

• Coding takes longer than traditional development because the verification step must 
be added. Programmers must truly understand their programs in order to verify the 
functions. 

• Understanding and verifying functions results in significantly fewer errors, which 
results in much less system test — an expensive part of development. 

• Overall result is lower cost and improved reliability. 

Since 1988, several projects have been developed in the SEL using the Cleanroom method- 
ology. To prepare developers for using the Cleanroom technique, a series of training courses 
were given. A pilot project was undertaken which proved to be very successful. Time to under- 
stand the method (from training until the start of the second Cleanroom project) was approxi- 
mately 26 months. Two follow-on Cleanroom projects were undertaken. A smaller in-house 
development was very successful, but a larger contracted project was not so successful. It was 
not clear whether problems on the larger project were due to scaling up of Cleanroom to larger 
tasks or to a lack of training and motivation of the development team on this project. Because 
of the differences that Cleanroom imposes on the development process, a fourth Cleanroom 
project is now underway for evaluation before declaring the technique “operational.” 

Compared to the SEL baseline process, it was clear that the Cleanroom development process 
was different (Figure 16). Design time and code reading grew significantly, while code writing 
and testing times all dropped. Defect rates improved (Figure 17) although productivity re- 
mained about the same using this new technology. The results of these studies are reported 
in more detail in [Basili 94], 
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5.2.4 Studies with Ada and OOD 


Some studies impose a great risk on the development organization. In such cases, 
experiments must be carefully controlled. The SEL evaluation of Ada was one such study. This 
experiment also shows the difficulty of trying to isolate single processes for evaluation. 

FORTRAN had always been the preferred programming language within NASA, but during the 
mid-1980s there was considerable interest in whether Ada should become their “language of 
choice.” The SEL had a baseline understanding of the FORTRAN development environment, 
but needed to develop a corresponding baseline for Ada. A controlled experiment was de- 
signed where the same onboard computer simulator would be developed in both Ada and 
FORTRAN in order to compare the two languages. 

In 1984, the GROSS project developed the operational FORTRAN simulator while a few 
months later an independent group, after first undergoing an intensive training program in the 
use of the language, developed the same simulator (GRODY) using Ada. 

The major result from this initial study was an improved understanding of the requirements 
used to specify NASA software. As the Ada simulator was being designed, it soon became ap- 
parent that the requirements document typically used in flight dynamics applications contained 
many functional design decisions inherent with an assumed use of FORTRAN. Based upon 
this finding, requirements for the simulator were respecified using an object-oriented approach 
indicating the use of OOD technology, data abstraction, and information hiding. Because of 
this redesign of the requirements, the SEL study encompassed both the applicability of Ada in 
the FDD and the use of OOD techniques. 

The GROSS-GRODY experiment was considered successful enough to try to use Ada on an 
actual mission, so several additional Ada projects were developed between 1987 and 1990 
(see Figure 18). As the SEL learned about Ada, the characteristics of Ada programs began to 
change: packages became smaller; use of generics rose; use of tasking dropped; and there 
was a greater use of the Ada typing mechanism as the programming staff became more famil- 
iar with the features of the language (Figure 19). 
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Figure 19: Maturing Use of Ada 


From these initial Ada studies, the SEL developed a model of Ada software development as 
compared to the traditional FORTRAN baseline: 

• First-time use of Ada resulted in a 30 percent increase in costs. 

• In general, line-by-line, Ada code is more expensive than FORTRAN code. 

• Reuse of Ada source code is higher than for FORTRAN, resulting in a 
decrease in program costs for Ada software. 

• Error rates were similar to error rates in FORTRAN. 
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Some of the attributes in Figure 1 8 are not unique to the Ada language but, rather, represent 
general OOD features. Given that, the knowledge obtained from these studies was packaged 
as the General Object-Oriented Software Development [Seidewitz 86] for application on mul- 
tiple projects in the environment. The result has been that FORTRAN programs, too, have 
greatly improved in their use of object-oriented techniques and in the reuse of components 
from system to system. Figure 20 shows the shortened schedules that have resulted from in- 
creases in reuse as object-oriented technology is increasingly employed on flight dynamics 
software. FORTRAN has continued to remain a competitive alternative to Ada as the technol- 
ogy has evolved. 


ADA FORTRAN 



EARLY RECENT EARLY RECENT 

(3 projects (3 projects (4 projects (3 projects 

1986-1990) 1991-1994) 1985-1990) 1991-1994) 


Figure 20: Reuse Shortened Project Duration 


5.2.5 Studies with IV&V 

Some process changes may not be appropriate for certain development organizations. The 
needs and goals must match the process. The following evaluation oflV& V was one such study. 

A study conducted in the mid-1980s is representative of the more formal experimentation pro- 
cess that the SEL typically uses. Much literature had been published indicating the value of 
using IV&V during the development of large software systems, so the SEL considered adopt- 
ing the methodology within the FDD production environment. However, before decisions were 
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made as to whether or not IV&V should become part of the standard process, several exper- 
iments were conducted to assess the cost, benefits, and compatibility of the technology for the 
SEL class of systems. 

Two experiments were designed to test IV&V on two major software development efforts. 
(These studies are described in detail in [Page 84].) The goal of using the technology was to 
drive software error rates down, while maintaining a relatively cost-effective development pro- 
cess. Each project was approximately 65 KSLOC and was typical of previous SEL tasks. The 
IV&V tasks had 3 full-time programmers and each project took approximately 16 months from 
design through acceptance. The initial expectations for these projects were 

• Increases in discovery of defects and quality of the operational software. 

• Decreases in design flaws, costs of correcting errors, and system test effort. 

• No changes in total defects reported. 


The requirements on the IV&V team were 

• Verify the requirements and design of the implemented system. 

• Perform separate system testing. Validate consistency of the system to its 
requirements. 

• Do not debug the programs, but report all anomalies. 


The results of the IV&V study are shown in Figure 21 and are summarized below: 

• Productivity dropped due to the increased costs of performing the IV&V 
function. 

• Errors found before system test were generally higher than the SEL average, 
but not excessively so. 

• IV&V did not significantly affect the overall error rate of SEL software. 

• IV&V errors cost about the same to fix as errors in previous SEL projects. 


While IV&V has been proposed in environments where it is critical to achieve a high degree of 
reliability, that situation was not apparent in the SEL environment. For the class of software 
that the SEL develops, IV&V was not deemed to be effective in improving either the reliability 
or overall cost of developing flight dynamics software. 
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5.2.6 Additional Studies 

In addition to the studies described, the SEL has experimented with numerous other 
technologies including testing coverage, code-reading techniques, computer-aided software 
engineering (CASE) technology, structured techniques, documentation approaches, defect 
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causal analysis, reuse approaches, and functional testing vs. structural testing, as well as 
many variations of these methodologies. For a complete list of SEL reports and publications 
see the Annotated Bibliography of SEL Literature [Morusiewicz 93]. 

Probably the most important lesson that has been derived from the studies is that specific 
techniques can help the overall goals of process improvement when appropriately selected 
and tailored. However, the most effective element of the improvement paradigm is the contin- 
uous analysis of the software business and the continuous expansion of the understanding of 
the software process and product. 

5.3 Packaging 

As the experiments provide additional insight into the most appropriate techniques, tools, and 
processes, results are identified and captured in the form of policies, standards, tools, and 
training for the development organization. The results of the understanding and analysis phas- 
es are captured and packaged for “reuse” by ensuing projects, so that they become part of the 
routine software business. 

For each of the studies conducted, the SEL analysts generate reports of results and conclu- 
sions. The reports may be papers for professional conferences, internal reports, or technical 
reports. Although the reports are valuable, the full value of the process analysis is felt when 
modifications and enhancements are made to the instruments that actually guide the way the 
development/maintenance organization carries out its business. These include standards, pol- 
icies, and training classes. The SEL development organization uses a standard set of policies 
that is updated on a periodic basis to reflect new experimentation results. For instance, the 
Manager’s Handbook for Software Development [Landis 90] reflects the process that the man- 
agers use on the flight dynamics systems. This handbook contains the models, guidelines, and 
acceptable processes expected to be applied on each of the development efforts. It is period- 
ically modified to reflect the completed studies of production projects. A full set of standards 
represents the changing process for this environment [Landis 92]. 

An important packaging concept is the infusion of technology in the form of support tools for 
use by project personnel. The SEL developed a project management tool called the Software 
Management Environment (SME). SME provides project managers access to the SEL data- 
base of previous project data and access to the baseline set of SEL process models. Using 
the SME, a manager can, for example, compare the growth rate of source programs or the 
growth rate of errors, or, using data from similar projects in the database, the manager can 
predict future activities on the current project. (For more details on the SME, see [Hendrick 
92].) Tools such as SME help institutionalize the packaging of the SEL process, because they 
do not require operational personnel to know all of the details of each model in order to use 
them to gain insight into their software projects. 
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SEL experiences have been packaged in a variety of formats. Figure 22 summarizes some of 
the recent SEL processes that have been understood, assessed, and then packaged. 


Assessed Processes 

Sample Study Results 

SEL Process 

Testing/Reading 

• Selby, Basili (85) 

L 1. Manager’s Handbook for Software 

Technologies 

• Card, Selby, McGarry (85) 

r Development 

Design Process 

• Card, Page, McGarry (85) 

b 2. Recommended Approach to 


• Seidewitz, Stark (87) 

Software Development 

IV&V 

• Page, McGarry, Card (85) 

f 3. Programmer’s Handbook for Flight 



k Dynamics Software Development 

Tools Usage 

• Valett, Hall, McGarry (84) 

► 

Cleanroom 

• Kouchakdjian, Green, Basili (89) 

L 4. Ada Style Guide 


• Green, Pajerski (91 ) 

V 


• Basili, Green (94) 

L 

Ada 

• Agresti, Bailey, Brophy (87) 

' 5. Cleanroom Process 


• McGarry, Agresti (88) 

L 


• Bailey, Waligora, Stark (94) 

r 

OOT 

• Seidewitz, Stark (87) (91 ) 

k 6. General Object-Oriented 


• Seidewitz (94) 

' Development 

CASE 

• McGarry (92) 

► 

Maintenance Modeling 

• Rombach, Ulery, Valett (92) 

k 7. SEL Training Plan 

Prototyping 

• Zelkowitz (87) 

r 

k 8. Approach to Software Cost 



W Estimation 

Cost Estimation 

• Condon et al (93) 

f 


Figure 22: SEL Packaging: SEL Software Process 


As part of the packaging process, the SEL has developed a standard set of training courses 
that is designed to provide all of the developers, managers, analysts, and database support 
staff with the information needed to function effectively in the FDD environment. Courses cov- 
er the SEL software process improvement concepts, software development methodology, 
software management approaches, standards, and organizational guidelines. This core set of 
courses reflects the experimental results, the process improvement approach, and, in general, 
all of the experiences of the SEL. These core courses are continually updated to reflect new 
and changing experiments within the SEL. 

In addition to the core courses, the SEL staff provides training in any technology, methodology, 
or process that is planned as part of a SEL study when the technology or process is unfamiliar 
to the development teams. For instance, extensive training was provided in Ada and OOT be- 
fore any attempt was made to apply these technologies on development projects. Other train- 
ing has included Cleanroom, inspections, and CASE. If the SEL staff does not possess the 
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skills or knowledge to teach the courses, appropriate instructors may be recruited from else- 
where in the organization or outside vendors may be contracted to provide the training. 

All SEL staff (managers, developers/maintainers, analysts, and database support) are re- 
quired to participate in the core set of training classes, while the staff from specific develop- 
ment experiments attend specialized training addressing the processes under study. 
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6 The SEL Impact 


The SEL has invested extensive time, energy, and resources in its efforts to better understand 
software process and its impact on software products. SEL studies have involved over 120 
projects and perhaps as many software technologies, ranging from development and manage- 
ment practices (e.g., structured technologies), to automation aids (e.g., CASE and develop- 
ment tools), to technologies that affect the full life cycle (e.g., Ada, OOD). The results of SEL 
experiments have been captured in a large collection of reports, papers, and documents. 
These publications are available to the public at no charge and are used as the foundation for 
extending the studies within the SEL. See [Morusiewicz 93] for a complete list of SEL-pub- 
lished and SEL-related literature. 

6.1 Impact on Product 

Individual studies often resulted in specific improvements on the project being studied, but 
many experiments resulted in no measurable improvements or even negative impact on the 
end product. The major goals of the SEL from the beginning called for significant overall im- 
provement in three product measures: 

• Decrease in the defect rate of delivered software. 

• Decrease in the cost of software to support similar missions. 

• Decrease in the average cycle time to produce mission support software. 

The additional measure of predictability has also always been a goal, but this is a more sub- 
jective measure and is more difficult to quantify. Detailed measures from the projects allowed 
the SEL staff to observe trends in the key measures over time and to analyze specific changes 
by comparing similar classes of software supporting similar classes of projects. In addition to 
the information that characterizes the measures identified above, additional data collected on 
all projects support more extensive comparisons of other product attributes. 

To determine the general impact of the sustained efforts of the SEL as measured against its 
major goals, comparisons were made between a group of projects developed between 1 985 
and 1 989 (the early baseline) and a group of similar projects developed between 1 990 and 
1993 (the current baseline). Projects were grouped based on size, mission complexity, mis- 
sion characteristics, language, and platform. Similar types of comparisons have been made 
over longer periods of time, as well as comparisons made on smaller sets of projects in varying 
classes. The goal of these analyses was to assess the return on investment derived from the 
process improvement program. This was measured as improvement in the end product in the 
three key measures: defects, cost, and cycle time. 
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The early baseline comprises 8 projects completed between 1985 and 1989 (see Figure 23). 
These projects were all ground-based attitude determination and simulation systems devel- 
oped on large IBM mainframe computers ranging in size from 50-150 KSLOC. All of these 
projects were considered successful in that they met mission dates and requirements within 
acceptable cost, and all of these projects applied some variation on the standard software pro- 
cess as part of SEL experimentation. The current SEL baseline comprises 7 projects complet- 
ed between 1 990-1 993 (see Figure 24). The analysis focused on a comparison of defect rates, 
cost, cycle time, and levels of reuse. Additionally, the reuse levels were studied carefully with 
the full expectation that there would be a correlation between higher reuse and lower cost and 
defect rates. 


Project 

% Reuse 

Cost 1 

(Staff-months) 

Reliability 

(Error/KDLOC) 

GROAGSS 

14 

381 

4.42 

COBEAGSS 

12 

348 

5.22 

GOESAGSS 

12 

261 

5.18 

UARSAGSS 

10 

675 

2.81 

GROSIM 

18 

79 

8.91 

COBSIM 

11 

39 

4.45 

GOESIM 

29 

96 

1.72 

UARSTELS 

35 

80 

2.96 


1 Mission cost = cost of telemetry simulator + cost of attitude ground support system. 

Figure 23: Early SEL Baseline (1985-1989) 


Project 

% Reuse 

Cost 1 

(Staff-months) 

Reliability 

(Error/KDLOC) 

EUVEAGSS 

81 

155 

1.22 

SAMPEX 

83 

77 

.76 

WINDPOLR 

18 

476 


EUVETELS 

96 

36 

.41 

SAMPEXTS 

95 

21 

.48 

POWITS 

69 

77 

2.39 

TOMSTELS 

97 

n/a 3 

.23 

FASTELS 

92 

n/a 3 

.69 


1 Mission cost = cost of telemetry simulator ♦ cost of attitude ground support system 

2 Clean room project excluded because errors are counted cfifferently. 

3 Ongoing projects; final costs unavailable. 


Figure 24: Current SEL Baseline (1990-1993) 
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(1990-1993) 


Figure 25: Impact on SEL Products (Reliability) 


Srtware cos was also compared between the 2 baselines. The mission cost is defined as the 

lion oHhe °1 3 Th 9M dynam ' CS software re 9 uired 10 support the flight project. An examina- 
“° f ,he ( selected missions from the 2 baselines revealed that while the total lines of code 

p oduced to support the specific missions has remained relatively close, the total mission cost 

of 357 staffToT i ^" "w T*. ‘ ‘ Vera9<i miSSi0 " the 8ady basellne ran 9 ad ,rom a low 
357 staff-months to a high of 755 staff-months with an average of 490 staff-months The 

urrantbasehne projects had costs ranging from a low of 98 staff-months to a high of 277 staff- 

The ^ontoint ri Ver39e °' 2 ' ° S,af, ' mon,hs ' R 9 ure 26 sbowa ' ba comparison the cost data. 

° 0St 03,1 a 6 3ttribUted '° inCreaS6S in bom P roduc «vrty and code 
reuse (Figure 27). This companson shows that the average cost per mission has decreased 

by over 50 percent over the 8-year period. aecreasea 


CMU/SEI-94-TR-22 


i 

E 

I 



Early 

(1985-1989) 


Current 

(1990-1993) 


Figure 26: Impact on SEL Products (Cost) 
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Figure 27: Impact on SEL Products (Reuse) 


Through the experimentation and emphasis on the reuse of software in the SEL, detailed data 
have been tracked that characterize the trends in the reuse of software. Although code reuse 
represents only one measure of software reuse, it is one of the more measurable and more 
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easily understood, so the SEL uses it to measure reuse in its environment. Code reuse is de- 
fined as the total lines of application code in components (compilable units) that have been 
taken in their entirety from a previously completed system or application library. Commercial 
off-the-shelf products and multiple use of a module within the same system are not included 
in the computations. 

In addition to examining the changes over recent years by comparing projects with similar 
characteristics, the long-term trends of reliability were examined for the full set of projects 
where accurate error data were available. Approximately 60 flight dynamics projects had ac- 
curate error data over the same phases of the life cycle. The error rate data were taken from 
these projects over the full lifetime of the SEL and were fit using a simple linear regression 
(shown in Figure 28). The data indicated that error rates decreased from approximately 7.5 
errors per KSLOC to approximately 1 error per KSLOC — an improvement of over 75 percent. 
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6.2 Impact on Process 

The SEL has reviewed in detail the process changes that have been tried and adopted over the 
lifetime of the improvement program. It would be satisfying to be able to point to a key technol- 
ogy or methodology change and to state that it had a direct, measurable link to a specific product 
improvement. However, it is difficult to isolate the impact of any one change in this environment. 
But the most significant changes that have been adopted can be identified by examining the pol- 
icies, training programs, and development approaches that today constitute the SELVFDD pro- 
cess. Although specific techniques or methodologies may have measurable impact on a class 
of projects, the significant improvement to the software development process is viewed at a 
higher level where the sustained, continuous incorporation of detailed techniques into higher 
level processes is the more significant payoff. The most significant process attributes that dis- 
tinguish the current SEL production environment from the environment of a decade ago include 
the following: 

• Process change has been infused as a standard business practice. 

The standards, policies, and training material all now contain elements of the 
continuous improvement approach to experimentation that has been 
promoted by the SEL. 

• Measurement is now our way of doing business. 

Measurement is no longer treated as an add-on to development. The 
measurement activity is as common a part of the software standards and 
policies as documenting software. It is expected, applied, and effective. 

• Change is now driven by product and process, not merely process alone. 

As the process improvement program has matured over the years, there has 
developed an equal concern about product attributes as well as process 
attributes. A set of product goals is always defined before process change is 
infused. Because of this, measures of product are as important as (and 
probably more important than) those of process. 

• Change is now bottom-up. 

Although process improvement analysts originally assumed that they could 
work independently from the developers, the years have brought the 
realization that change must be guided by development-project experience. 

Direct input from developers as well as measures extracted from 
development activities are key factors in change. 

• “People-oriented” technologies are emphasized rather than automation. 

The process changes that have been most effective are those that leverage 
the thinking ability of the developers. These include reviews, inspections, 
Cleanroom techniques, management practices, and independent testing 
techniques, all of which are driven by disciplined activities of the 
programmers/managers. Automation techniques have sometimes provided 
improvement, but not to the extent that people-driven approaches have. 
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6.3 Cost of Change 

The benefits of the process improvement efforts are well substantiated by looking at the mea- 
sures of software cost, error rates, and cycle time — all goals of the organization as change was 
being implemented. Not only has the SEL traced the detailed software measures throughout 
the 17-year lifetime, but expenditures incurred due to the efforts of process change have also 
been tracked quite closely. The SEL investment in process change activities can be divided 
into 3 significant areas: 

• Project overhead 

• Data handling, archiving, and technical support 

• Process analysis 

The total investment that the SEL has made in the improvement effort has been approximately 
1 0 percent of the total software development cost in the FDD. Project overhead represents 
costs incurred due to developers attending training (in new processes), completing data col- 
lection forms, participating in interviews, and providing detailed additional information request- 
ed by the analysts. This overhead for data collection and process change is extremely small. 
It is now nearly impossible to measure except in the cases of very large process changes, 
such as using a new language (longer training, meetings, etc.). For projects participating in the 
routine process improvement efforts, the impact is less than 1 percent of the total software 
cost. A successful process improvement program does not require a large perturbation or cost 
to the development organization. 

Data archiving and repository activities require a larger investment. Not only must measures 
be collected from the developers, but there must be a smooth process of data quality assur- 
ance, archiving, and reporting. This function of the SEL has cost approximately three percent 
of the total development budget. That is, the database support element has cost approximately 
eight million dollars over the lifetime of the SEL. This figure includes purchase and design of 
database management systems and distribution of SEL literature as well. 

The analysis activity has been the most costly of all the expenditures. The responsibilities of 
the analysts include setting goals, defining experiments, interpreting measurement data, train- 
ing the development/maintenance staff, developing standards and policies, and tailoring pro- 
cesses for particular needs. The analysts must provide refined processes to the development 
organization along with rationale of why one process is more appropriate than another. They 
must design and then provide any required training to the development organization. 

SEL activities have cost approximately $25 million in an organization that has spent approxi- 
mately $250 million on software development and maintenance. There is no empirical evi- 
dence to indicate whether this 1:10 ratio would persist when process improvement is 
expanded to a larger organization. Very preliminary data from SEL experience in carrying out 
a broader NASA-wide program indicate that project overhead would remain low, and that the 


CMU/SEI-94-TR-22 


49 


data repository could function with 15 persons to support development organizations that are 
orders of magnitude larger than the FDD. Figure 29 shows the observed process improvement 
cost based on the experiences of the FDD organization. 
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Figure 29: ROI for Process Improvement in the FDD 


6.4 Impact on Other Organizations 

The SEL has operated in one domain at NASA/GSFC for 17 years and the measures of 
change have been apparent. In addition to the impact of the process activities on this local or- 
ganization, the overall concept of change and improvement has been adopted by other orga- 
nizations at GSFC and across NASA. 
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Systems Engineering and Analysis Support (SEAS) Contract 

CSC is the prime contractor on the SEAS contract (valued at $1 billion over 10 years) which 
supports NASA with a staff of nearly 1400 developers and analysts. This project has expanded 
the application of SEL concepts beyond the FDD. Drawing on the SEL's experience in 
documenting the methodology used in the flight dynamics environment and on CSC’s general 
corporate methodology, SEAS has developed and documented a system development 
methodology for use across the entire contract and a set of standards and procedures to help 
staff members apply it. 

Established guidelines for quantitative management are also used on the SEAS contract. Rec- 
ognizing the importance of quantitative management, as shown by the SEL's success, SEAS 
has packaged its experiences in this area in a data collection, analysis, and reporting hand- 
book to be used by all the managers. 

The SEAS organization also offers a training program, modeled after the SEL’s example. An 
established, required training program exists for all engineers, developers, testers, integrators, 
and managers working on the contract. This required training is designed to ensure consistent 
understanding and application of the SEAS System Development Methodology and of the 
quantitative management techniques used in the SEAS environment. 

Earth Observing System (EOS) Core System Development Organization 
The EOS Core System, one of GSFC’s largest development efforts (totaling $760 million over 
20 years), has adapted the SEL process improvement concept. The EOS program is managed 
by Hughes Applied Information System. SEL personnel are currently consulting with Hughes 
to help guide the development of a process improvement program for the EOS organization. 

NASA Software Engineering Program 

In 1991, a software engineering program modeled on the SEL’s concepts was initiated by 
NASA Headquarters for application across the Agency. To date, GSFC and four other NASA 
centers have participated in the program. The first phase, baselining of all NASA software, is 
ongoing and scheduled for completion in December of 1 994. 
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7 Summary 


Over the course of its history, the SEL has pioneered a measurement-based paradigm for soft- 
ware process improvement. Long before it became fashionable to think of software develop- 
ment in terms of an “engineering discipline,” the founders of the SEL had the vision to 
recognize that software developers follow a process, and that if that process could be captured 
qualitatively and measured quantitatively, a baseline could be established against which im- 
provements could be measured. What has evolved since that time is a unique “bottom-up” ap- 
proach to software process improvement that is driven by a solid understanding of an 
organization’s environment (process and product characteristics), coupled with clearly defined 
organizational goals for process improvement. This approach eventually leads to a steady 
state that involves a continuous cycle of understanding the current baseline, introducing new 
methods and techniques in a controlled fashion and measuring their impact, refining those 
techniques deemed effective, and packaging the resulting process changes for infusion back 
into the organization’s standard process. 

Over the years, the SEL has validated this paradigm and measured its impact in the FDD en- 
vironment. It has executed experiments and studies ranging from small-scale comparisons of 
code reading and testing techniques to long-term evaluations of major technologies, such as 
Ada, object orientation, and Cleanroom. Each of these studies resulted in some modification 
to the standard flight dynamics software development process, usually in the incorporation of 
key concepts from the technologies studied tailored to maximize their leverage within the en- 
vironment. As documented earlier in this report, the resulting impact on product measures has 
been significant, and when the costs of implementing the program are tallied, the return on in- 
vestment is substantial. 

The SEL is committed to sharing its findings as to the improvements it has witnessed in flight 
dynamics software development and what techniques have or have not made an impact. More 
importantly, however, the SEL is equally committed to sharing the process improvement para- 
digm it has forged, and all of the lessons it has learned along the way. Many of these results are 
published in software engineering journals and presented at major international conferences. In 
addition, the SEL sponsors an annual Software Engineering Workshop, with paper sessions, 
panels, and tutorials, that draws an audience of over 400 software engineering practitioners 
from around the world. The SEL has also captured its research results and process models in 
a series of documents, many of which have been sent to thousands of individuals and organi- 
zations. As discussed in Chapter 6, the SEL’s concepts have been adapted and adopted by oth- 
er organizations, including Hughes Applied Information System, NASA Headquarters, and the 
SEL’s partner in industry, Computer Sciences Corporation. Thus, the SEL has had, and contin- 
ues to have, a significant impact on the software engineering community. 
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Looking to the future, current efforts are just getting underway to establish SEL-based process 
improvement programs at various other U.S. Government agencies. There is also considerable 
interest from several sectors in examining the expansion issues related to applying the SEL par- 
adigm beyond local environments to higher organizational levels. The SEL anticipates being ac- 
tively involved in supporting these efforts. It also plans to continue supporting the process 
improvement goals of the FDD organization. Most importantly, perhaps, is the SEL’s continued 
commitment to sharing its processes, findings, and lessons learned throughout the software 
community, and by so doing, advancing the state-of-the-practice in software engineering. 
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Appendix A Sample SEL Experiment Plan 

SEL Representative Study Plan for SOHOTELS 
October 11,1 993 

A.1 Project Description 

The Solar and Heliospheric Observatory Telemetry Simulator (SOHOTELS) software devel- 
opment project will provide simulated telemetry and engineering data for use in testing the 
SOHO Attitude Ground Support System (AGSS). SOHOTELS is being developed by a team 
of four GSFC personnel in Ada on the STL VAX 8820. The project is reusing design, code, 
and data files from several previous projects but primarily from the Solar, Anomalous, and 
Magnetospheric Particle Explorer Telemetry Simulator (SAMPEXTS). 

The SOHOTELS team held a combined preliminary design review (PDR) and critical design 
review (CDR) in April 1993. In their detailed design document, the SOHOTELS team stated 
the following goals for the development effort: 

• To maximize reuse of existing code. 

• Where reuse is not possible, to develop code that will be as reusable as 
possible. 

• To make sure performance does not suffer when code is reused. 

A.2 Key Facts 

SOHOTELS is being implemented in three builds so that it can be used to generate data for 
the early phases of the AGSS (which is a Cleanroom project). Build development and inde- 
pendent acceptance testing are being conducted in parallel. At present, the test team has fin- 
ished testing SOHOTELS Build 1 . The development team expects to complete Build 2 and 
deliver it to the independent test team by the end of the week. 

SOHOTELS consists of six subsystems. As of June, the estimated total number of compo- 
nents was 435, of which 396 (91 percent) have currently been completed. Total SLOC for SO- 
HOTELS was estimated at 67.6 KSLOC, with 46.6 KSLOC of code to be reused verbatim and 
15.7 KSLOC to be reused with modifications. As of September 13, 1993, there were 65.4 
KSLOC in the SOHOTELS system, or 97 percent of the estimated total. 

The SOHOTELS task leader is currently re-estimating the size of the system because SO- 
HOTELS will be more complex than was originally predicted. The new estimates will include 
SLOC for the schema files that are being developed. 
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The phase start dates for SOHOTELS are: 


September 9, 1992 

Requirements Definition 

October 3, 1992 

Design 

May 1, 1993 

Code and Unit Test 

June 26, 1993 

Acceptance Test 

May 7, 1993 

Cleanup 


A.3 Goals of the Study 

The study goals for SOHOTELS are 

• To validate the SEL’s recommended tailoring of the development life cycle for 
high-reuse Ada projects. 

• To refine SEL models of high-reuse software development projects in Ada, 
specifically: 

• effort (per DLOC, by phase and by activity); 

• schedule (duration for telemetry simulators and by phase); 

• errors (number per KSLOC/DLOC); 

• classes of errors (e.g., initialization errors, data errors); and 

• growth in schedule estimates and size estimates (from initial estimates to 
completion and from PDR/CDR to completion). 

A.4 Approach 

The following steps will be taken to accomplish the study goals: 

• Understand which of the standard development processes are being 
followed and which have been tailored for the SOHOTELS project. Ensure 
that information is entered into the SEL database that will allow SOHOTELS 
data to be correctly interpreted in light of this tailoring. 

• Analyze project/build characteristics, effort and schedule estimates, effort 
and schedule actuals, and error data on a monthly basis while development 
is ongoing. 

• At project completion, plot the effort, schedule, error rate, and estimate data. 
Compare these plots with current SEL models and with plots from other high- 
reuse projects in Ada. Compare and contrast the error-class data with data 
from FORTRAN projects, from Ada projects with low reuse, and from other 
high-reuse Ada projects. 
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A.5 Data Collection 


To address these study goals, the following standard set of SEL data for Ada projects will be 
collected: 

• Size, effort, and schedule estimates (Project Estimates Forms). 

• Weekly development effort (Personnel Resources Forms). 

• Growth data (Component Origination Forms and SEL librarians). 

• Change and error data (Change Report Forms and SEL librarians). 
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Appendix B FDD Projects 

The following tables are taken from the SEL Cost and Schedule Estimation Study Report [Con- 
don 93]. They are included here to provide an overview of the basic characteristics of the 
many FDD projects studied by the SEL over the years. Table B-2 summarizes system size 
and reuse rates for the same projects. 
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PASA 

AGSS 

F 

05/76 — 09/77 

69 

15760 

ISEEB 

AGSS 

F 

10/76 — 09/77 

50 

15262 

AEM 

AGSS 

F 

02/77 — 03/78 

57 

12588 

SEAS AT 

AGSS 

F 

04/77 — 04/78 

54 

14508 

ISEEC 

AGSS 

F 

08/77 — 05/78 

38 

5792 

SMM 

AGSS 

F 

04/78 — 10/79 

76 

14371 

MAGSAT 

AGSS 

F 

06/78 — 08/79 

62 

15122 

FOXPRO 

AGSS 

F 

02/79 — 10/79 

36 

2521 

DEA 

AGSS 

F 

09/79 — 06/81 

89 

19475 

DEB 

AGSS 

F 

09/79 — 05/81 

83 

17997 

DESIM 

TS 

F 

09/79 — 10/80 

56 

4466 

ERBS 

AGSS 

F 

05/82 — 04/84 

97 

49476 

DERBY 

DS 

F 

07/82 — 11/83 

72 

18352 

GROSS 

DS 

F 

12/84 — 10/87 

145 

15334 

GRODY 

DS 

A 

09/85 — 10/88 

160 

23244 

COBEDS 

DS 

F 

12/84 — 01/87 

105 

12005 

ASP 

AGSS 

F 

01/85 — 09/86 

87 

17057 

GROAGSS 

AGSS 

F 

08/85 — 03/89 

188 

54755 

GROSIM 

TS 

F 

08/85 — 08/87 

100 

11463 

COBSIM 

TS 

F 

01/86 — 08/87 

82 

6106 

COBEAGSS 

AGSS 

F 

06/86 — 09/88 

116 

49931 

GOADA 

DS 

A 

06/87 — 04/90 

149 

28056 

GOFOR 

DS 

F 

06/87 — 09/89 

119 

12804 

GOESAGSS 

AGSS 

F 

08/87 — 11/89 

115 

37806 

GOESIM 

TS 

A 

09/87 — 07/89 

99 

13658 

UARSAGSS 

AGSS 2 

F 

11/87 — 09/90 

147 

89514 

ACME 

AGSS 2 

F 

01/88 — 09/90 

137 

7965 

UARS_2 

AGSS 2 

F 

N/A 

N/A 

97479 

UARSDSIM 

DS 

F 

01/88 — 06/90 

128 

17976 

UARSTELS 

TS 

A 

02/88 — 12/89 

94 

11526 

EUVEAGSS 

AGSS 

F 

10/88 — 09/90 

102 

21658 

EUVE 2 3 

AGSS 

F 

N/A 

N/A 

21658 

EUVETELS 

TS 

A 

10/88 — 05/90 

83 

4727 

EUVEDSIM 

DS 

A 

10/88 — 09/90 

121* 

20775 4 

SAMPEXTS 

TS 

A 

03/90 — 03/91 

48 

2516 

SAMPEX 

AGSS 5 

F 

03/90 — 11/91 

85 

4598 

SAMPEXTP 

AGSS 5 

F 

03/90 — 11/91 

87 

6772 

SAMPEX_2 

AGSS 5 

F 

N/A 

N/A 

11370 

POWITS 

TS 

A 

03/90 — 05/92 

111 

11695 

Abbreviations: A Ada DS Dynamics si mulataor TS Telemetry simulator 

AGSS Attitude Ground F FORTRAN simulator 

Support System 

1 Design phase through acceptance test phase. 

2 tf !? UARS sat0,,i,e was developed as two projects. One project, containing the majority of the AGSS code and functionality, was called simply 
UARSAGSS >and was developed by CSC. The other project, containing two utilities (CFADS and STAR ID), was called ACME and was developed in-house 
by GSFC. When referring to the total size or effort of the two combined projects, this study uses the name UARS_2. 

3 The EUVE AGSS was developed as a single project, and the EUVEAGSS account in the SEl database includes all hours spent on this AGSS. In recording 
the lines of code in the EUVEAGSS account, however, the SEL database did not include the ACME lines of code, all of which were borrowed from the ACME 
project and reused verbatim in the EUVE AGSS. When referring to the size or productivity of the total EUVE AGSS, this study uses the name EUVE 2 The 
values for effort and schedule duration do not vary between EUVE AGSS and EUVE_2. 

Duration adjusted by +15% and Effort adjusted by +10% because EUVEDSIM did not have an acceptance test phase. These values are consistent with those 
of the Ada Size Study Report. 

5 The AGSS for the SAMPEX satellite was developed as two projects. The telemetry processor part, called SAMPEXTP, was developed in-house by GSFC. 
The other project, containing the majority of the AGSS code and functionality, was called simply SAMPEX and was developed by CSC When referrinq to the 
total size or effort of the two combined projects this study uses the name SAMPEX_2. 

Includes technical staff and technical management hours for preproject through cleanup phases. Does not include support staff-hours (project management 
librarians, secretaries, technical publications). 

Table B-1: Projects Studied 
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Project 


Newly Written 


Extensively Modified 


Slightly Modified 


Reused 

Verbatim 


PAS 

84729 

0 

20041 

7098 

ISEEB 

43955 

0 

3506 

7776 

AEM 

45345 

0 

4673 

893 

SEASAT 

49316 

0 

4252 

21825 

ISEEC 

20075 

0 

6727 

48618 

SMM 

76883 

0 

5652 

2834 

MAGSAT 

61950 

0 

14297 

13266 

FOXPRO 

5354 

0 

1323 

2449 

DEA 

45004 

0 

9705 

12616 

DEB 

44644 

0 

8606 

13016 


DESIM 

ERBS 

DERBY 

GROSS 

GRODY 

COBEDS 

ASP 

GROAGSS 

GROSIM 

COBSIM 

COBEAGSS 


14873 

137739 

37137 

33196 

123935 

26986 

70951 

194169 

31775 

45825 

141084 


3493 

1143 


9982 


1342 

16017 


5767 

3901 

8574 

3037 

7363 


18133 

4294 

1156 

13647 


385 

15635 

4549 

6441 

146 

2556 

10483 

14109 

2881 

4494 

7934 


GOADA 


109807 


12496 


41750 


7049 


GOFOR 


22175 


2867 


6671 


5330 


GOESAGSS 


106834 


6377 


9779 


5869 


GOESIM 


59783 


5784 


15078 


11450 


UARSAGSS 


260382 


9340 


21536 


11868 


ACME 


34902 


UARS 2 


295284 


9340 


21536 


11868 


UARSDSIM 


63861 


17476 


20710 


4399 


UARSTELS 


38327 


6114 


12163 


11544 


EUVEAGSS 


41552 


13597 


14844 


179016 


EUVE 2 


41552 


13597 


14844 


213918 


EUVETELS 


2161 


371 


5573 


58591 


EUVEDSIM 


20859 


36248 


87415 


39495 


SAMPEXTS 


3301 


6120 


52026 


SAMPEX 


10590 


1631 


1282 


141006 


SAMPEXTP 


15899 


1920 


1777 


36 


SAMPEX_2 

POWITS 


26489 

12974 


3551 

7980 


3059 

20878 


141042 

26275 


Table B-2: Detailed Line-of-Code Data 
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